System and method for fusing three-dimensional shape data on distorted images without correcting for distortion

ABSTRACT

A system and method for intra-operatively providing a surgeon with visual evaluations of possible surgical outcomes ahead of time, and generating simulated data, includes a medical imaging camera, a registration device for registering data to a physical space, and to the medical imaging camera, and a fusion mechanism for fusing the data and the images to generate simulated data, without correcting for distortion. The simulated data (e.g., such as augmented X-ray images) is natural and easy for a surgeon to interpret. In an exemplary implementation, the system preferably includes a data processor which receives a three-dimensional surgical plan or three-dimensional plan of therapy delivery, one or a plurality of two-dimensional intra-operative images, a three-dimensional model of pre-operative data, registration data, and image calibration data. The data processor produces one or a plurality of simulated post-operative images, without correcting for distortion, by integrating a projection of a three-dimensional model of pre-operative data onto one or a plurality of two-dimensional intra-operative images.

CROSS-REFERENCE TO RELATED APPLICATION

[0001] The present application is related to U.S. patent applicationSer. No. 09/299,643, filed on Apr. 27, 1999, to Gueziec et al., entitled“SYSTEM AND METHOD FOR INTRA-OPERATIVE, IMAGE-BASED, INTERACTIVEVERIFICATION OF A PRE-OPERATIVE SURGICAL PLAN” having IBM Docket No.YO999-095, assigned to the present assignee, and incorporated herein byreference.

BACKGROUND OF THE INVENTION

[0002] 1. Field of the Invention

[0003] The present invention generally relates to robotics and medicalimaging techniques and, more particularly to robotically-assistedsurgical systems and other devices incorporating methods for registeringimage data (both pre-operative and intra-operative) to physical spaceand for providing feedback, and in particular visual feedback, to theclinician.

[0004] 2. Description of the Related Art

[0005] Computers are increasingly used to plan complex surgeries byanalyzing pre-operative Computed Tomography (CT) or Magnetic ResonanceImaging (MRI) scans of a patient.

[0006] To execute the surgical plan, it is important to accurately alignor register the three-dimensional pre-operative and intra-operative datato an actual location of the patient's anatomical features of interestduring surgery.

[0007] One conventional technique for performing this type ofregistration is to attach a stereo-tactic frame or fiducial markers tothe patient, and to precisely locate the frame or markers prior to andduring surgery.

[0008] For example, in the case of a surgery involving a patient'sfemur, a conventional registration protocol includes implanting threemetallic markers or pins in the patient's femur (e.g., one proximally inthe trochanter and two distally in the condyles, near the knee).However, the insertion of the pins requires minor surgery. A CT-scanimage of the patient is subsequently acquired. By analyzing the CT data,the surgeon decides upon the size and location of the implant that bestfits the patient's anatomy. During surgery, the metallic pins areexposed at the hip and knee. The patient's leg is attached to a surgicalrobot device that then must locate the exposed pins. A registration, orcoordinate transformation from CT space to robot space, is computedusing the locations of the three pins as a Cartesian frame. The accuracyof this registration has been measured to be better than one millimeter.This conventional registration protocol is described in U.S. Pat. No.5,299,288 entitled “IMAGE-DIRECTED ROBOTIC SYSTEM FOR PRECISE ROBOTICSURGERY INCLUDING REDUNDANT CONSISTENCY CHECKING” by Glassman et al.,and incorporated herein by reference.

[0009] However, using such pins as markers is not always desirable, asthey may cause significant patient discomfort, and the required surgicalprocedure to insert and subsequently remove the pins is inconvenient andcostly to the patient.

[0010] An alternative registration technique is to perform anatomy-basedregistration that uses anatomical features of the patient (e.g.,generally bone features), as markers for registration. Conventionalmethods for performing anatomy-based registration are described in“Registration of Head CT Images to Physical Space Using a WeightedCombination of Points and Surfaces” by Herring et al., in IEEETransactions on Medical Imaging, Vol. 17, No 5, pages 753-761, 1998 andin U.S. patent application Ser. No. 08/936,935 (YO997-322) entitled“METHODS AND APPARATUS FOR REGISTERING CT-SCAN DATA TO MULTIPLEFLUOROSCOPIC IMAGES”, filed on Sep. 27, 1997 by A Gueziec et al., eachof which is herein incorporated by reference in its entirety.

[0011] Once the registration has been performed, it is important toprovide the clinician with means to assess the registration, allowinghim or her to validate, reject or improve the registration (and thesurgical plan). A system and method for advising a surgeon is describedin U.S. Pat. No. 5,445,166, entitled “SYSTEM FOR ADVISING A SURGEON”, byTaylor, which is herein incorporated by reference in its entirety.Taylor describes a system for guiding the motions of a robot, or of apositioning device controlled by motors, and teaches how audio feedbackand force feedback can be provided to a surgeon. Taylor also describes avisual adviser allowing comparison of the surgical plan with itsexecution. The system taught by Taylor optionally uses a camera at theend of a surgical instrument that sends an image to the graphicsadapter, optionally mixed with graphics output of the computer.

[0012] A conventional technique for simulating a post-operative X-rayimage is described in “An Overview of Computer-Integrated Surgery at theIBM T. J. Watson Research Center” by Taylor et al., in IBM Journal ofResearch, 1996, which is herein incorporated by reference in itsentirety.

[0013] Thus, conventional techniques are useful for registeringthree-dimensional pre-operative and intra-operative data to an actuallocation of anatomical features of interest during surgery, and toprovide advice to the surgeon. However, none of the conventionaltechniques teaches how to simulate a post-operative condition dependingupon the registration of image data to physical space, by fusingintra-operative images with registered pre-operative data, andgenerating new images.

[0014] In Taylor et al., the simulated post-operative X-ray image isgenerated using only pre-operative CT (Computed Tomography) data.Herring et al. do not teach how to evaluate the registration accuracyintra-operatively.

[0015] Although Glassman et al.'s and Taylor's systems compare asurgical plan and its execution, neither Glassman et al. nor Taylorteaches how to simulate the outcome of a surgical plan prior to theactual execution of the plan. With Taylor's system, a surgeon can takecorrective measures to minimize the effects of a wrongful execution ofthe plan, but cannot make a decision before any execution of the planand therefore cannot prevent all errors before they occur.

[0016] Further, the information produced by Taylor's system for advisinga surgeon is not represented in the form of conventional medical media(e.g., such as X-ray images) and requires an extra burden on the surgeonin order to interpret and evaluate this information.

[0017] Thus, it is believed that conventional techniques do not exist(or at the very least are inadequate) for (a) providing the surgeon withpost-operative evaluations prior to surgery, that are obtained bymerging intra-operative image data and pre-operative data, and (b)presenting such evaluations in a standard clinical fashion (e.g., suchas augmented X-ray images) that is natural for a surgeon to interpret.

[0018] Other problems of the conventional systems and methods includethe limited availability of 2-D/3-D registration methods in conventionalart systems for advising a surgeon.

[0019] In another conventional system, as described in theabove-mentioned U.S. patent application Ser. No. 09/299,643, thegeometric distortion of an X-ray image is always corrected. This isproblematic because a clinician or surgeon is used to seeing theunmodified image (e.g., an image with distortion). That is, as apractical matter, surgeons generally are not familiar with seeing themodified image. Surgeons are used to interpreting the unmodified images.Further, such a correction may cause image degradation or blurring dueto the reformatting of the image. Additionally, slower and more complexprocessing results from the image correction process.

SUMMARY OF THE INVENTION

[0020] In view of the foregoing and other problems of the conventionalmethods and structures, an object of the present invention is to providea method and structure for intra-operatively providing the surgeon withvisual evaluations of possible surgical outcomes ahead of time, theevaluations being obtained by merging intra-operative image data andpre-operative data, and presented in a standard clinical fashion (e.g.,such as augmented X-ray images) that is natural and easy for a surgeonto interpret.

[0021] Another object of the present invention is to provide a systemand method for fusing three-dimensional shape data on distorted imageswithout correcting for distortion.

[0022] Yet another object of the present invention is to provide asystem and method for assisting the surgeon in improving an inaccurateregistration of a pre-operative surgical plan to a physical space of anoperating room.

[0023] Still another object of the present invention is to provide animproved robotically assisted surgical system that also provides visualpost-operative evaluations.

[0024] A further object of the present invention is to provide animproved robotically-assisted surgical system that includes a system forassisting the surgeon in improving a registration.

[0025] Another object of this invention is to provide an improvedrobotically assisted surgical system that includes a system forpreventing surgical errors caused by internal failure of the robot'scalibration system.

[0026] In a first aspect of the present invention, a system is providedfor fusing three-dimensional shape data on distorted images withoutcorrecting for distortion.

[0027] The inventive system preferably includes a data processor. Thedata processor takes as inputs a three-dimensional surgical plan orthree-dimensional plan of therapy delivery, one or a plurality oftwo-dimensional intra-operative images, a three-dimensional model ofpre-operative data, registration data, and image calibration data.

[0028] The data processor produces one or a plurality of simulatedpost-operative images, by integrating a projection of athree-dimensional model of pre-operative data onto one or a plurality oftwo-dimensional intra-operative images, without correcting for anydistortion in the images.

[0029] The data processor optionally receives an input from a surgeon ora clinician. The input preferably includes a set of constraints on thesurgical plan or plan of therapy delivery. The data processor preferablyoptimizes the surgical plan or plan of therapy delivery using theconstraints.

[0030] In another aspect of the present invention, a system (and method)for generating simulated data, includes a medical imaging camera forgenerating images, a registration device for registering data to aphysical space, and to the medical imaging camera, and a fusion(integration) mechanism for fusing (integrating) the data and theimages, without correcting for distortion to generate simulated data.

[0031] In yet another aspect of the invention, a method of fusingthree-dimensional image data on distorted images, includes receiving apotentially distorted image, calibrating the potentially distortedimage, based on the calibration, computing an apparent contour of thethree-dimensional shape of the potentially distorted image, for eachpixel of the image, determining a ray in 3-dimensional space andcomputing a distance from the ray to the apparent contour, andselectively adjusting a pixel value of the potentially distorted imagebased on the distance.

[0032] In yet another aspect of the invention, a signal-bearing mediumis provided for storing a program for performing the method of theinvention. Other aspects of the invention are also set forth below.

[0033] With the invention, the surgeon is provided with intra-operativevisual evaluations of possible surgical outcomes in advance, with theevaluations being obtained by merging intra-operative image data andpre-operative data. Such evaluations are presented in a standardclinical fashion that is natural and easy for a surgeon to interpret.Further, the inventive system compares several registration methods ofpre-operative data to the physical space of the operating room.

[0034] Moreover, the invention assists the surgeon in improving aninaccurate registration of a pre-operative surgical plan to the physicalspace. Additionally, the system can be robotically-assisted and canprovide visual post-operative evaluations.

[0035] Additionally, in the robotically-assisted implementation of theinventive system, surgical errors, caused by internal failure of therobot's calibration system, can be prevented.

[0036] Further, with the invention, a clinician or surgeon can viewimages in the manner that they are accustomed. That is, the clinician orsurgeon can view the unmodified image (e.g., an image with distortion),in the manner with which they are familiar. Thus, the surgeons cancontinue to interpret the unmodified images, as is customary. Further,since no correction is performed as in the conventional system andmethods, no image degradation or blurring results from such imagecorrection and reformatting of the image. Additionally, processing speedis not decreased, and similarly processing resources are not increasedsince the processing of the method of the present invention is lesscomplex than that of the conventional systems and methods.

BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The foregoing and other purposes, aspects and advantages will bebetter understood from the following detailed description of preferredembodiments of the invention with reference to the drawings, in which:

[0038]FIG. 1 is a block diagram of a preferred embodiment of a systemaccording to the present invention;

[0039]FIG. 2 is a flow chart showing an overview of a process togenerate a post-operative simulation;

[0040]FIG. 3 is a flow chart showing an overview of a process for fusingthree-dimensional shape data on distorted image without correcting fordistortion;

[0041]FIG. 4 illustrates a technique of finding a center of perspectiveof a three-dimensional shape (e.g., implant) to the image and a raydestination for each pixel of the image;

[0042]FIG. 5 is a schematic diagram for explaining the decomposition ofa shape into visible and invisible sub-shapes (e.g., triangles) whichare separated by apparent contours;

[0043]FIG. 6 is a schematic diagram for explaining a “current edge” anda “next edge” in the process of obtaining the apparent contours;

[0044] FIGS. 7A-7E illustrate a pre-operative model and FIGS. 7F-7Hillustrate distorted images; and

[0045]FIG. 8 illustrates a storage medium 800 for storing steps of theprogram for fusing three-dimensional shape data on a distorted imagewithout correcting for distortion.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS OF THE INVENTION

[0046] Referring now to the drawings, and more particularly to FIGS.1-8, there is shown a preferred embodiment of the method and structureaccording to the present invention.

[0047] Generally, the present invention resides in a system and methodto intra-operatively provide the surgeon with visual evaluations ofpossible surgical outcomes ahead of time, the evaluations being obtainedby merging intra-operative image data and pre-operative data, and beingpresented in a standard clinical fashion (such as augmented X-rayimages) that is natural and easy for a surgeon to interpret.

[0048] The present invention differs from the invention in U.S. Pat.application Ser. No. 09/299,643 by omitting the step of correcting thegeometric distortion of the X-ray image in the method of U.S. patentapplication Ser. No. 09/299,643 (e.g., step 2040 in FIG. 2 thereof) andother processing as described below.

[0049] That is, the step of correcting the geometric distortion of theX-ray image is omitted between the step of obtaining registrationinformation from the X-ray image to the pre-operative CT image and thestep of using the registration and calibration information to projectpre-operative data on the X-ray image.

[0050] A novel aspect of the present invention is to allowintra-operative manipulation of a model (e.g., such as a CAD model of animplant) as opposed to a real object (e.g., such as a cutter of asurgical robot as in Taylor's system).

[0051] Referring to FIG. 1, a system 1000 according to the presentinvention uses a two-dimensional intra-operative image 1010 (e.g., atwo-dimensional X-ray or other type of image) and a three-dimensionalshape of a prosthetic implant 1020, and comprises a data processor 1040.The pre-operative image (e.g., of the shape of the implant with respectto anatomical features) may be obtained by an X-ray, computed tomography(CT) scanner, whereas the intra-operative image(s) may be obtained by atwo-dimensional (2D) X-ray camera.

[0052] The data processor 1040 receives the image 1010 and the shape1020, as well as registration data 1050 and a surgical plan 1060. Theregistration data registers the shape 1020 with the camera used foracquiring the image 1010. An example of registration process producingregistration data 1050 is provided in the above-mentioned U.S. patentapplication Ser. No. 08/936,935, previously incorporated by reference.

[0053] A typical example of the surgical plan 1060 is a planned type,orientation and position of an implant relative to anatomical structuresin a pre-operative CT scan. Another example of the surgical plan 1060 isthe planned type, orientation and position of an implant relative toco-registered intra-operative X-ray images of anatomical structures.

[0054] Image calibration data 1070 is also input to the data processor.The data processor 1040 produces a simulated post-operative image 1030.Image 1030 may be presented visually to the surgeon on a display 1035.That is, the post-operative simulation (e.g., data which preferablyincludes an image such as a 2-dimensional image) may be displayed on anyof a cathode ray tube (CRT), liquid crystal display (LCD), or the like.

[0055] Referring now to FIGS. 2-6, the operation of the presentinvention will be described hereinbelow.

[0056]FIG. 2 is a flow chart illustrating how a post-operativesimulation can be generated using a method 2000 according to the presentinvention.

[0057] In Step 2010, an image (e.g., an X-ray image or otherintra-operative image 1010 as shown in FIG. 1) is capturedintra-operatively. Conventional methods for capturing an X-ray imageinclude using a frame grabber connected to the video output of aconventional fluoroscope. Fluoroscopes are manufactured by many medicalimaging equipment manufacturers. An example of a fluoroscope is theZiehm Exposcop Plus® System (Exposcop Plus is a trademark of the ZiehmCorporation). Another method for capturing an X-ray imageintra-operatively is to use an X-ray flat panel detector. An example ofan X-ray flat panel detector is the FlashScan 30®. FlashScan 30 is atrademark of the DPIX Corporation.

[0058] Then, in Step 2020, a geometric calibration of the X-ray image isperformed. Geometric calibration is preferably performed using theteachings of the above-mentioned U.S. patent application Ser. No.08/936,935.

[0059] In Step 2030, X-ray and pre-operative CT data are registered(e.g., this data represents the registration data 1050 of FIG. 1). Apreferred method for registering X-ray and pre-operative CT data isdescribed in the above-mentioned U.S. patent application Ser. No.08/936,935.

[0060] Then, in Step 2040, without correcting for the geometricdistortion of the X-ray image as in the above mentioned method in U.S.patent application Ser. No. 09/299,643, the registration and calibrationare used to project pre-operative data such as a three-dimensional shapeof an implant (e.g., shape 1020 in FIG. 1) onto the X-ray image. Theresult is the simulated post-operative image 1030 in FIG. 1.

[0061] Essentially, when bypassing the step of reformatting the image(e.g., correcting the image for distortion), the shape may besuperimposed on the image that is still distorted. The following steps,as shown in FIG. 3, are equivalent to applying the same distortionpresent in the image when projecting the three-dimensional shape ontoit. The shape is projected onto the image and the projection processincorporates a distortion process, which becomes complex.

[0062] In Step 3010, an image (e.g., X-ray image) that is potentiallydistorted, is calibrated. To perform calibration, a system is used forassociating a center of perspective 4010 (e.g., as shown in FIG. 4) tothe image and for determining a “ray destination” 4020 for each pixel ofthe image. For example, the system described in the above-mentioned U.S.patent application Ser. No. 08/936,935 could be used for this purpose.

[0063] In Step 3020, a set of three-dimensional apparent contours iscomputed knowing the center of perspective 4010 (e.g., FIG. 4) and thethree-dimensional shape 4030 is decomposed into sub-shapes (e.g.,triangles as shown in greater detail in FIGS. 5 and 6 discussed furtherbelow).

[0064] The processing for decomposing the apparent contour intotriangles is further described in U.S. patent application Ser. No.09/236,688, entitled “SYSTEM AND METHOD FOR FINDING THE DISTANCE FROM AMOVING QUERY POINT TO THE CLOSEST POINT ON ONE OR MORE CONVEX ORNON-CONVEX SHAPES”, by A. Gueziec, filed on Jan. 25, 1999 as IBM DocketY0999-024, incorporated herein by reference in its entirety. Othershapes be used instead of or in addition to triangles. For example,polygonal shapes could be used instead of or in addition to triangles,as would be known by one of ordinary skill in the art taking the presentapplication as a whole.

[0065] Given the center of perspective (possibly very far such as, forexample, 1 meter for X-rays; of course, such a distance depends on thefocal length of the imaging camera and could be more for another imagingcamera source) from the surface, three-dimensional apparent contours aredefined and extracted as follows. It is noted that it is possible tohave only one apparent contour depending upon the shape involved in theviewing direction. Generally, a complex curve has visible and invisibleedges as shown in FIGS. 5 and 6.

[0066] That is, as shown in FIGS. 4-6, for each surface triangle, the“viewing direction” is defined as the vector originating from the centerof perspective to the triangle centroid.

[0067] If the triangle normal (e.g., defined by the cross product ofordered oriented triangle edges, as generally known by one of ordinaryskill in the art of computer graphics) makes an obtuse angle with theviewing direction, the triangle is considered “visible”. Otherwise, itis considered “invisible”.

[0068] Surface apparent contours are a subset of surface edges, suchthat the triangle on one side of the edge is visible and the triangle onthe other side of the edge is invisible.

[0069] Referring to FIG. 5, an example of a visible triangle andinvisible triangle are shown. That is, a visible triangle 5010 and aninvisible triangle 5020 are illustrated in FIG. 5. The apparent contoursare such that the edges are linked to form (non-planar) closed polygonalcurves in three dimensions.

[0070] To build the apparent contours, all edges, belonging to anyapparent contour using the criterion defined above, are identified, andsuch edges are added to a list (e.g., a table or register with variousedges input thereto). The edges are oriented such that the visibletriangle is on the left side of the edge, thus defining an edge origin5030 and an edge destination 5040.

[0071] Then, the following process is iterated. For clarity, the readeris referred to FIG. 6.

[0072] That is, first, the first edge in the list is taken, and a newapparent contour is created starting with that edge (e.g., step 1).

[0073] Then, the apparent contour, containing that edge, is completed asfollows (e.g., step 2). Starting from the destination of a current edge6010, a next edge 6020 is determined. The triangles, incident to thedestination vertex in a counter-clockwise fashion (e.g., just aconvention; a clockwise direction could alternatively be employed), arevisited, and the first edge is determined that belongs to the list ofapparent contour edges. This is necessary because there may be severalsuch edges. Step 2 above is re-applied (e.g., reiterated) until the nextedge is the same as the first edge that was processed in step 1.

[0074] In a third step, all the edges forming that contour from the listof apparent contour edges, are removed. Then, steps (1) to (3) arere-applied until the list of apparent contour edges is empty.

[0075] Then, in Step 3030 of Step 3 for each pixel of the potentiallydistorted image, the corresponding ray (e.g., line) from the center ofperspective is determined, and the distance to the apparent contours iscomputed. The distance from a given line in three-dimensions to anapparent contour, which is a particular type of curve inthree-dimensions may be preferably computed as follows.

[0076] First, the teachings of the above-mentioned U.S. patentapplication Ser. No. 09/236,688, incorporated herein by reference, maybe applied.

[0077] In the above-mentioned patent application, one of the steps usesa method for computing the distance from a point to a line segment inthree-dimensional space. This method should be replaced with a methodfor computing the distance from a line in three-dimensions to a linesegment in three-dimensions. Various conventional methods may be usedfor this purpose, that are known to those skilled in the art. Such amethod is described on p.10 of “Locally Toleranced SurfaceSimplification”, A. Gueziec, IEEE Transactions on Visualization andComputer Graphics, Vol 5, No. 2, 1999.

[0078] Finally, in Step 3040, the distance ray-shape 4040 that wasdetermined in the previous step, is used to update the pixel value.Various correspondences between distance values and pixel values may beused for this purpose. The correspondence used in the above-mentionedU.S. patent application Ser. No. 09/299,643, incorporated herein byreference, may be used for this purpose.

[0079] Thereafter, a process for validating, rejecting or improving asurgical plan using post-operative simulations, as described in theabove-mentioned U.S. patent application Ser. No. 09/299,643,incorporated herein by reference.

[0080] In an implementation of the above process, FIGS. 7A-7E illustratea pre-operative model, in which FIG. 7A illustrates a CT-based proximalfemur model, FIG. 7B illustrates an implant CAD model, FIG. 7Cillustrates a simplified CAD Model of a fiducial pin (e.g., exemplarydimensions of 8 mm diameter by 10 mm), FIG. 7D illustrates a femur andpin model registered in CT space, and FIG. 7E illustrates an implant andpin model registered in CT space. FIG. 7F-7H illustrates distortedimages. That is, the projection of the shapes is distorted according tothe image distortion model. FIG. 7F shows superimposing the proximal pin(e.g., anatomy-based registration) and that the pin model is longer thanthe physical pin. FIG. 7G show superimposing femur and implant models,wherein FIG. 7G is a marker based registration, whereas FIG. 7H shows ananatomy.

[0081] Thus, in the invention, for a given X-ray image, usingcalibration information, first a center of perspective is determinedwhich is used to compute silhouette curves of the implant model (asexplained in U.S. patent application Ser. No. 09/299,643). Then, themethod works independently of whether distortion-corrected images ordistorted images are produced.

[0082] That is, for each pixel of the X-ray image (original imagepixels, or rectified image pixels), an X-ray path from the (u, v) (grid)coordinates corresponding to the pixel and the center of perspective aredetermined. Then, the distance is computed from each X-ray path to theimplant or other shape as discussed in U.S. patent application Ser. No.09/299,643.

[0083] Finally, the computed distances are converted to gray-scalevalues. Various methods can be used to do this. To produce the images,the following mapping was used: if the distance was less than 0.05 mm, agray-scale value of 0 was used, otherwise, if the distance was less than0.1 mm, a gray-scale value of 30 was used, otherwise, if the distancewas less than 0.2 mm, a gray-scale value of 60 was used, and otherwise,no change to the existing gray-scale value was done. This method avoids“aliasing” in the implant outline (i.e., “staircase” effects in theresulting line drawings). One advantage of using distances to silhouettecurves is that the resulting projection of the implant shows only theprojected silhouette, which is sufficient to precisely indicate theposition of the implant, but does not obscure any of the anatomy.

[0084] As shown in FIG. 8, in addition to the hardware and processenvironment described above, a different aspect of the inventionincludes a computer-implemented method for fusing three-dimensionalshape data on distorted images without correcting for distortion, asdescribed above. As an example, this method may be implemented in theparticular hardware environment discussed above.

[0085] Such a method may be implemented, for example, by operating aCPU, to execute a sequence of machine-readable instructions. Theseinstructions may reside in various types of signal-bearing media.

[0086] Thus, this aspect of the present invention is directed to aprogrammed product, comprising signal-bearing media tangibly embodying aprogram of machine-readable instructions executable by a digital dataprocessor incorporating the CPU and hardware above, to perform a methodof fusing three-dimensional shape data on distorted images withoutcorrecting for distortion.

[0087] This signal-bearing media may include, for example, a randomaccess memory (RAM) contained within the CPU, as represented by afast-access storage, for example. Alternatively, the instructions may becontained in another signal-bearing media, such as a magnetic datastorage diskette 800 (FIG. 8), directly or indirectly accessible by theCPU.

[0088] Whether contained in the diskette 800, the computer/CPU, orelsewhere, the instructions may be stored on a variety ofmachine-readable data storage media, such as DASD storage (e.g., aconventional “hard drive” or a RAID array), magnetic tape, electronicread-only memory (e.g., ROM, EPROM, or EEPROM), an optical storagedevice (e.g. CD-ROM, WORM, DVD, digital optical tape, etc.), paper“punch” cards, or other suitable signal-bearing media includingtransmission media such as digital and analog communication links andwireless. In an illustrative embodiment of the invention, themachine-readable instructions may comprise software object code,compiled from a language such as “C”, etc.

[0089] While the invention has been described in terms of severalpreferred embodiments, those skilled in the art will recognize that theinvention can be practiced with modification within the spirit and scopeof the appended claims.

[0090] It is noted that the present invention can be implemented in manyapplications.

[0091] For example, the invention can be used in orthopedic surgery(e.g., such as total hip replacement surgery, revision total hipreplacement surgery, spine surgery, etc.). In one implementation, thepre-operative images typically are three-dimensional CT images or MRI(Magnetic Resonance Imaging) images, and the intra-operative imagestypically are X-ray fluoroscopy images. A three-dimensionalpre-operative plan (e.g., such as planning position of a prostheticimplant with respect to the surrounding bony anatomy) may be integratedonto one or several two-dimensional X-ray images to provide the surgeonwith images to evaluate a potential surgical outcome.

[0092] The present invention also can be used in treating cancer byradio-therapy. Conventional radio-therapy delivery devices include animaging device (e.g., producing “portal” images), whereas the presentinvention can be used to project a three-dimensional radio-therapy planonto two-dimensional images produced by the imaging device, therebyproviding the clinician with a mechanism and technique to evaluate theaccuracy with which the therapy will be delivered.

[0093] The present invention also can be used in brain surgery, in whichcase the pre-operative images typically may be three-dimensional CT orMRI images, and the intra-operative images typically may be X-rayimages. A three-dimensional surgical plan (e.g., such as planning theremoval of a tumor of a specified shape and location relatively to thesurrounding imaged anatomy) may be integrated onto one or severaltwo-dimensional X-ray images to provide the surgeon with images toevaluate a potential surgical outcome.

[0094] The present invention also can be used in craniofacial surgery.In such a case, the pre-operative images typically would bethree-dimensional CT or MRI images, and the intra-operative imagestypically would be X-ray images. A three-dimensional surgical plantypically would involve osteotomies and the relocation of bone fragmentsto correct some physical deformities. A robotic device would be used tomanipulate bone fragments. The three-dimensional plan would beintegrated onto one or several two-dimensional X-ray images to providethe surgeon with images to evaluate a potential surgical outcome, and inparticular to compare the resulting images with X-ray images of normalindividuals, or to evaluate that the execution of the plan will becorrect.

What is claimed is:
 1. A system for generating simulated data,comprising: a medical imaging camera for generating images, said imagesincluding distorted images; a registration device for registering datato a physical space, and to said medical imaging camera; and a fusionmechanism for fusing said data and said distorted images, to generatesimulated data.
 2. The system according to claim 1 , further comprising:another medical imaging camera for collecting said data, and whereinsaid medical imaging camera collects said images.
 3. The systemaccording to claim 1 , wherein said data comprises pre-operative imagesand said images comprise intra-operative images.
 4. The system accordingto claim 1 , wherein said data comprises a two-dimensional image.
 5. Thesystem according to claim 1 , wherein said simulated data comprisessimulated post-operative images.
 6. The system according to claim 1 ,wherein said fusion mechanism generates said simulated data while asurgery is being performed, and without correcting for distortion ofsaid distorted images.
 7. The system as in claim 3 , wherein thepre-operative data comprises data of a surgical plan including aposition of a component, and a three-dimensional shape of saidcomponent.
 8. The system as in claim 7 , wherein said componentcomprises an implant for a patient.
 9. The system as in claim 2 ,wherein said another imaging camera comprises an X-ray, computedtomography (CT) scanner.
 10. The system as in claim 1 , wherein saidmedical imaging camera comprises a two-dimensional (2D) X-ray camera.11. The system as in claim 1 , wherein said fusion mechanism comprises adata processor.
 12. A system for fusing three-dimensional shape data ondistorted images, comprising: an imaging camera; a registration devicefor registering data to a physical space, and to the imaging camera; anda fusion mechanism for fusing said data and distorted intra-operativeimages to generate simulated data without correcting for distortion. 13.A system for providing intra-operative visual evaluations of potentialsurgical outcomes, using medical images, comprising: a first medicalimaging camera for collecting pre-operative images; a second medicalimaging camera for collecting distorted intra-operative images, while asurgery is being performed; a registration mechanism for registeringsaid pre-operative images and other pre-operative data to a physicalspace, and to said second medical imaging camera; and a fusion mechanismfor fusing said pre-operative data and said distorted intra-operativeimages, without correcting for distortion, to generate simulatedpost-operative images.
 14. A method of fusing three-dimensional imagedata on an image, comprising: receiving a potentially distorted image;computing an apparent contour of said three-dimensional image data onsaid potentially distorted image; for each pixel of the image,determining one of a ray in three-dimensional space and computing adistance from the ray to the apparent contour; and selectively adjustinga pixel value of said potentially distorted image based on saiddistance.
 15. The method according to claim 14 , further comprising:calibrating said potentially distorted image, wherein said computing isbased on said potentially distorted image having been calibrated. 16.The method according to claim 15 , wherein said calibrating comprises:associating a center of perspective to the image and a ray destinationfor each pixel of the potentially distorted image.
 17. The methodaccording to claim 16 , wherein said computing comprises: decomposingsaid three-dimensional image data into predetermined sub-shapes; andcomputing a set of three-dimensional apparent contours based on saidcenter of perspective and the three-dimensional image data beingdecomposed into said predetermined sub-shapes.
 18. The method accordingto claim 17 , wherein said sub-shapes comprise sub-shapes having one ofa triangular shape and a polygonal shape.
 19. The method according toclaim 16 , wherein said computing further comprises: based on saidcenter of perspective, defining and extracting a three-dimensionalapparent contour.
 20. The method according to claim 19 , wherein saiddefining and extracting comprises: for each surface sub-shape, defininga viewing direction as a vector originating from the center ofperspective to a centroid of the sub-shape.
 21. The method according toclaim 20 , wherein if the sub-shape normal, as defined by a crossproduct of ordered oriented sub-shape edges, makes an obtuse angle withthe viewing direction, the sub-shape is considered visible, wherein asurface apparent contour is a subset of surface edges, such that asub-shape on one side of the edge is visible and a sub-shape on anotherside of the edge is invisible, said apparent contour having edges linkedto form non-planar polygonal curves in three dimensions.
 22. The methodaccording to claim 21 , wherein said forming of said apparent contourcomprises: identifying edges belonging to any apparent contour, andadding said edges to a list, wherein said edges are oriented such that avisible sub-shape is on a predetermined side of the edge, thus definingan edge origin and an edge destination.
 23. The method according toclaim 22 , wherein said forming of said apparent contour furthercomprises: based on a first edge in the list, creating a new apparentcontour starting with said first edge; and completing said apparentcontour, containing said first edge.
 24. The method according to claim23 , wherein said completing said apparent contours comprises: startingfrom the destination of the current edge, completing a next edge,wherein sub-shapes incident to a destination vertex in acounter-clockwise fashion, are visited, and the first edge is determinedthat belongs to the list of apparent contour edges; and reapplying saidcompleting until a next edge is the same as the first edge that wasprocessed previously.
 25. The method according to claim 24 , whereinsaid forming said apparent contour further comprises: removing all theedges forming that contour from the list of apparent contour edges. 26.The method according to claim 25 , wherein said forming said apparentcontour further comprises: reapplying said creating a new apparentcontour, completing the apparent contour, and said removing until thelist of apparent contour edges is empty.
 27. The method according toclaim 16 , wherein said determining comprises: for each pixel of thepotentially distorted image, determining the corresponding ray from thecenter of perspective, and computing the distance to the apparentcontour.
 28. The method according to claim 27 , wherein computing saiddistance from a given line in three-dimensions to an apparent contour,comprises: computing the distance from a line in three-dimensions to aline segment in three-dimensions.
 29. The method according to claim 16 ,wherein said adjusting said pixel value comprises: updating the pixelvalue with a distance ray-shape that was determined.
 30. The methodaccording to claim 15 , further comprising: fusing saidthree-dimensional image data with said potentially distorted image byintegrating a two-dimensional projection of a silhouette of athree-dimensional implant model in an X-ray image.
 31. The methodaccording to claim 30 , wherein said fusing uses a calibration of theX-ray image, to determine a center of perspective whose locationrepresents an estimate of a location of an X-ray source, said center ofperspective being used to compute silhouette curves on thethree-dimensional implant model.
 32. The method according to claim 31 ,wherein said silhouette curves are such that rays emanating from thecenter of perspective and tangent to the three-dimensional model meetthe three-dimensional implant model on a silhouette curve.
 33. Themethod according to claim 16 , further comprising: fusing by projectinga silhouette curve of said data by considering in turn each new pixel,determining a line in three-dimensions corresponding to that pixel byimage calibration, computing a distance from a line to the silhouettecurve, and assigning a pixel gray-scale value depending on the distance.34. The method according to claim 33 , wherein said fusing comprisesassigning pixel gray-scale values corresponding to a distance, whereinif the distance is less than a first predetermined value, then thegray-scale value is set to a first predetermined number.
 35. The methodaccording to claim 34 , wherein if the distance is less than a secondpredetermined value, then the gray-scale value is set to a secondpredetermined number larger than said first predetermined number. 36.The method according to claim 35 , wherein if the distance is less thana third predetermined value greater than said first and secondpredetermined values, then the gray-scale value is set to a thirdpredetermined number larger than said first and second predeterminednumbers.
 37. The method according to claim 36 , wherein if the distanceis greater than or equal to said third predetermined value, then thegray-scale value is not modified for projecting the silhouette curves.38. A method of generating simulated post-operative data, comprising:collecting pre-operative data and collecting potentially distortedintra-operative images; registering said pre-operative data to aphysical space, and to a medical imaging camera; and fusing saidpre-operative data and said intra-operative images to generate simulatedpost-operative data, without correcting for distortion.
 39. The methodof claim 38 , further comprising: calibrating the medical imagingcamera, wherein the pre-operative data comprises data of a surgical planincluding a position of a component, and a three-dimensional shape ofsaid component.
 40. The method of claim 38 , wherein said component ofsaid surgical plan includes an implant position.
 41. An apparatus forfusing three-dimensional image data on an image, comprising: means forreceiving a potentially distorted image; a processor for computing anapparent contour of said three-dimensional image data on saidpotentially distorted image; means for determining, for each pixel ofthe image, one of a ray in three-dimensional space and computing adistance from the ray to the apparent contour; and means for selectivelyadjusting a pixel value of said potentially distorted image based onsaid distance.
 42. A signal-bearing medium tangibly embodying a programof machine-readable instructions executable by a digital processingapparatus to perform a method for computer-implemented fusing ofthree-dimensional image data on a distorted image without correcting fordistortion, comprising: computing an apparent contour ofthree-dimensional image data of a potentially distorted image; for eachpixel of the image, determining a ray in three-dimensional space andcomputing a distance from the ray to the apparent contour; andselectively adjusting a pixel value of said potentially distorted imagebased on said distance.
 43. A signal-bearing medium tangibly embodying aprogram of machine-readable instructions executable by a digitalprocessing apparatus to perform a method for computer-implementedgenerating of simulated post-operative data, comprising: collectingpre-operative data and collecting potentially distorted intra-operativeimages; registering said pre-operative data to a physical space, and toa medical imaging camera; and fusing said pre-operative data and saidintra-operative images to generate simulated post-operative data,without correcting for distortion.